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PREFACE 


Policy  analysts  have  always  lived  with  great  uncertainty.  We  have 
long  had  tools  for  handling  some  of  that  uncertainty.  But  now  that 
policy  analysis  has  been  around  for  a  while,  work  in  many  fields  has 
fallen  into  stereotyped  problem  formulations  and  analytical  ap¬ 
proaches.  In  particular,  treatments  of  uncertainty  are  typically  quite 
incomplete  and  often  conceptually  wrong.  This  report  argues  that 
these  shortcomings  produce  pervasive  systematic  biases  in  analyses. 
Problem  formulations  suppress  uncertainty  about  the  world  and 
ignore  the  capacity  of  the  analyzed  systems  to  adapt  to  the 
unanticipated.  System  design  ignores  opportunities  to  hedge  or 
respond  instead  of  predicting.  System  evaluation  discounts  strategies 
that  hedge  or  respond. 

Policy  analyses  change  in  specific  ways  if  their  problems  are  formu¬ 
lated  with  a  fuller  recognition  of  uncertainty.  Management  capabil¬ 
ity — the  ability  to  respond  and  adapt — becomes  a  resource,  with  a 
cost  and  an  effectiveness,  to  which  funds  can  be  allocated  in  competi¬ 
tion  with  more  familiar  uses  of  funds.  Analytical  tools  change  to 
incorporate  options  for  management.  Given  that  uncertainty  is 
pervasive  and  generally  cannot  be  counted  completely,  the  logic  that 
connects  the  analysis  to  the  world  changes  as  well:  Our  models  are 
not  necessarily  adequate  representations  of  reality. 

Although  its  main  example  is  an  Air  Force  logistics  problem,  this 
report  is  intended  for  policy  analysts  in  all  fields.  It  should  also  be  of 
interest  to  operations  researchers,  statisticians,  and  researchers  in 
other  quantitative  fields. 

This  research  drew  support  from  RAND's  three  federally  funded 
research  and  development  centers: 

•  Arroyo  Center 

•  National  Defense  Research  Institute 

•  Project  AIR  FORCE 

The  work  was  performed  under  the  project  entitled  "Logistics 
Monographs." 


SUMMARY 


If  events  were  predictable,  policy  analysis  would  be  simple.  For 
example,  if  Air  Force  planners  knew  which  parts  would  fail,  and 
where  and  when  they  would  fail,  it  would  be  straightforward  to 
schedule  repair  and  procurement  of  spares  and  repair  capacity.  But 
planners  do  not  know  what  will  fail  or  where  or  when  and  must  plan 
accordingly.  Every  policy  analytic  problem  shares  this  uncertainty  to 
some  degree.  The  common  mode  of  analysis  omits  crucial  sources  of 
uncertainty  and  overlooks  the  ability  of  systems  to  respond  to  the  un¬ 
expected.  These  two  omissions  systematically  bias  design  and  eval¬ 
uation  of  systems. 

The  process  of  analysis  is  broken  into  five  stages:  (1)  setting  the 
context,  (2)  choosing  alternatives,  (3)  predicting  consequences,  (4) 
valuing  outcomes,  and  (5)  making  a  choice.  This  critique  of  the  com¬ 
mon  approach  applies  mostly  to  the  first  four  stages,  which  rely  on  a 
three-part  representation  of  the  actual  problem: 

•  a  tasking  posited  for  the  system  under  study,  consisting  of  re¬ 
sponsibilities  assigned  to  the  system  and  conditions  under 
which  they  will  be  discharged, 

•  a  mechanism  that  turns  the  tasking  into  a  load  on  the  system, 
and 

•  the  system  under  study,  which  is  taxed  by  the  load  and  con¬ 
sumes  and  manages  resources  to  service  it.  For  emphasis,  we 
call  it  the  managed  system. 

Typically,  policymakers  and  analysts  manipulate  resources  sup¬ 
plied  to  the  managed  system.  For  example,  in  Air  Force  spares  and 
repair  requirements  calculations,  one  tasking — a  hypothetical  war, 
part  failure  rates,  and  repair  and  resupply  times — is  derived  from 
higher-level  guidance.  The  mechanism  consists  mostly  of  models  that 
relate  flying  hours  to  numbers  of  parts  needing  repair.  The  managed 
system — the  part  repair  and  resupply  system — is  modeled  without 
constraints  on  repair  or  transportation  capacity.  The  requirement 
computation  determines  how  much  repair  capacity  and  stock  to  buy 
for  the  managed  system  without  regard  for  the  way  the  repair  capac¬ 
ity  and  stock  are  managed. 


The  common  mode  of  analysis  omits  sources  of  uncertainty  from 
each  part  of  the  representation.  One  such  source  i3  statistical  uncer¬ 
tainty:  variability  observed  in  repeatable  phenomena.  Another 
source  of  uncertainty  arises  in  phenomena  that  are  not  repeatable, 
not  observed,  or  both:  state-of-the-world  uncertainty.  It  comes  in 
three  main  types:  extrapolation  away  from  observed  conditions  and 
behavior  of  people  inside  or  outside  the  mechanism  or  managed  sys¬ 
tem. 

In  Air  Force  spare  parts  and  repair  requirement  calculations,  some 
variation  in  part  failures  is  represented,  but  substantially  less  than  is 
observed  in  peacetime.  State-of-the-world  uncertainty  is  ignored. 
This  latter  includes  uncertainty  about: 

•  The  tasking:  Which  war  will  be  fought,  under  what  condi¬ 
tions? 

•  The  mechanism:  Can  peacetime  failure  behavior  be  extrapo¬ 
lated  to  wartime? 

•  The  managed  system:  Adaptations  made  by  the  repair  and 
resupply  system  are  ignored;  what  capability  will  they  pro¬ 
vide  in  wartime,  and  at  what  cost? 

This  example  is  typical  of  nonmilitary  analyses  as  well  as  of 
military  analyses. 

If  things  are  so  bad,  why  aren’t  more  systems  hopelessly  bogged 
down?  Because  mechanisms  and  managed  systems  are  not  inert,  as 
they  are  modeled,  but  adaptive.  In  the  Air  Force  example,  more  and 
less  informal  adaptations  in  the  repair  and  resupply  system  can  shift 
resources  toward  critical  parts.  Adaptive  behavior  provides  a  capabil¬ 
ity  to  function  in  the  face  of  uncertainty,  and  analysts  may  help  man¬ 
agers  more  by  devising  (and  evaluating)  ways  for  them  to  adapt,  in¬ 
stead  of  by  trying  to  help  them  predict  the  unpredictable. 

This  approach  has  been  explored  to  some  extent  with  a  study  of 
some  Air  Force  spare  parts  initiatives,  called  CLOUT  (Coupling 
Logistics  to  Operations  to  meet  Uncertainty  and  the  Threat).  This 
study  added  to  the  usual  analyses: 

•  Uncertainty,  in  the  form  of  actual  peacetime  variability  in 
part  failures  and  base  attack  damage; 

•  Management  adaptations,  such  as  sharing  of  repair  resources 
among  bases. 
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The  CLOUT  analysis  indicated  that  these  measures  could  produce 
a  substantial  payoff  in  aircraft  availability  in  combat  theaters. 

The  critique  here  has  implications  for  the  common  mode  of  policy 
analysis,  mostly  about  design  and  evaluation  of  systems,  which  com¬ 
prise  the  first  four  of  the  five  stages  of  policy  analysis.  The  implica¬ 
tions  include: 

•  System  Design  (Stage  2,  Choosing  Alternatives):  Systems 
need  strategies  for  coping  with  uncertainty.  We  identify  sev¬ 
eral  strategies  as  a  source  of  ideas  for  specific  design  problems 
and  to  illustrate  the  importance  of  understanding  the  relative 
value  of  management  (adaptiveness)  and  goods  (ampleness). 

•  System  Evaluation  (Stage  3,  Predicting  Consequences,  and 
Stage  4,  Valuing  Outcomes):  Models  can  represent  more  sta¬ 
tistical  uncertainty,  and  analyses  can  allow  for  more  state-of- 
the  world  uncertainty  in  a  sensitivity  analysis  by  considering 
variants  on  the  tasking.  But  often  these  two  methods  are  not 
enough  to  connect  the  models  and  sensitivity  analysis  to  the 
world:  They  can  capture  more  uncertainty,  but  not  enough. 
In  such  cases,  other  kinds  of  arguments  are  needed  to  make  a 
clean  logical  connection  between  the  analysis  and  the  world. 
We  examine  two:  prototyping  (or  similar  operational  tests) 
and  a  fortiori  arguments.  Prototyping  is  a  superb  method  but 
is  available  for  only  a  limited  class  of  problems.  For  other  sit¬ 
uations,  the  a  fortiori  argument  may  be  available.  The 
CLOUT  analyses  are  an  example.  The  model  used  in  those 
analyses  treats  part  failures  as  more  predictable  than  they 
really  are,  and  allows  fewer  management  adaptations  than 
will  probably  be  available.  In  this  world,  the  CLOUT  initia¬ 
tives  are  preferable  to  current  nonadaptive  methods.  In  an 
actual  war,  conditions  will  probably  be  more  favorable  to 
CLOUT,  so  a  fortiori,  CLOUT  appears  to  prevail. 

•  Setting  the  Tasking  (Stage  1,  Setting  the  Context):  Analyses 
must  reflect  uncertainty  about  the  tasking  by  considering  a 
range  of  taskings.  This  does  not  mean,  for  example,  that 
budget  requests  must  be  ranges  instead  of  single  numbers;  it 
does  mean  that  single  budget  or  requirement  numbers  must 
be  evaluated  against  a  range  of  taskings  instead  of  a  single 
tasking.  We  conjecture  that  it  will  be  fruitful  to  explore 
choosing  taskings  to  drive  a  fortiori  arguments  like  the  above. 
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L  INTRODUCTION 


BACKGROUND 

If  events  were  predictable,  policy  analysis  would  be  simple.  For 
example,  if  Air  Force  planners  knew  which  aircraft  parts  would  fail 
and  where  and  when  they  would  fail,  it  would  be  straightforward  to 
schedule  repair  of  parts,  to  buy  parts  and  repair  capability,  and  to 
make  budgets.  But  nobody  can  accurately  predict  which  parts  will 
fail,  when  they  will  fail,  or  where  they  will  fail,  even  in  peacetime. 
The  wars  for  which  we  prepare  are  also  shrouded  in  uncertainty. 
How  much  will  our  aircraft  be  called  on  to  fly,  and  on  what  kinds  of 
missions?  How  much  damage  will  our  repair  assets  sustain?  Will 
parts  fail  in  wartime  as  they  do  in  peacetime?  Again,  no  one  knows. 

The  Air  Force  captures  some  of  this  uncertainty  in  its  computa¬ 
tions  and  analyses  about  spare  parts  and  repair.  For  example,  uncer¬ 
tainty  about  part  failures  is  described  by  simple  probability  dis¬ 
tributions  with  a  simple  extrapolation  to  wartime.  However,  much 
uncertainty  is  omitted  from  all  the  Air  Force’s  computations  and 
analyses.  For  example,  apart  from  the  simple  extrapolation  just 
mentioned,  all  wartime  uncertainty  is  omitted  from  computations 
used  for  budgets,  requirements  estimates,  and  operating  the  spares 
and  repair  system. 

Not  coincidentally,  these  same  computations  generally  use  a  rudi¬ 
mentary  model  to  represent  the  system  ;hat  manages  spare  parts  and 
repair,  if  they  represent  it  at  all.  Thus,  repair  and  transportation  ca¬ 
pacity  are  usually  assumed  to  be  infinite,  and  repairs  are  assumed  to 
be  scheduled  based  on  a  first-come-first-served  rule.  As  a  result,  the 
system’s  capacity  to  adapt  and  respond  to  the  unexpected  is  simplified 
away,  and  the  implications  of  that  capacity  are  lost. 

OBJECTIVE 

The  common  mode  of  analysis,  as  illustrated  by  the  Air  Force  ex¬ 
ample  just  mentioned,  omits  crucial  sources  of  uncertainty  and  over¬ 
looks  the  ability  of  systems  to  respond  to  the  unexpected.  These  flaws 
introduce  a  systematic  bias  into  analyses  and,  thus,  into  the  systems 
that  are  motivated  and  evaluated  by  those  analyses.  Analyses  serve 
many  purposes:  Some  are  done  repeatedly  to  drive  operations;  others 
are  done  once,  such  as  for  purchases;  and  others  are  performed  for 
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still  different  purposes.  The  omissions  play  out  differently  in  the  dif¬ 
ferent  roles  that  analyses  fill,  and  our  critique  has  implications  for  all 
these  roles.  Further,  although  our  main  example  will  be  the  case  of 
Air  Force  spare  parts  and  repair,  the  report’s  implications  apply  to  all 
areas  in  which  policy  analysis  is  done.  We  indicate  this  with  brief  ex¬ 
amples  from  munitions  requirements  computations,  disposal  of  high- 
level  radioactive  waste,  and  fire  department  management,  among 
others. 

ORGANIZATION  OF  THE  REPORT 

Section  II  describes  the  common  mode  of  policy  analysis  and  identi¬ 
fies  the  two  main  shortcomings  raised  in  this  section.  The  next  two 
sections  discuss  them  in  turn.  Section  III  categorizes  some  varieties 
of  uncertainty  relevant  to  policy  analysts  and  gives  some  examples  of 
how  they  are  commonly  represented.  The  reader  is  then  asked  to  re¬ 
call  that  our  military  organizations  are  not  hopelessly  bogged  down, 
and  Sec.  IV  argues  that  this  is  so  because  managers  are  not  as  inert 
as  our  models  of  them:  Managers  respond  to  unpredicted  circum¬ 
stances.  Their  responses  provide  system  capability,  the  costs  and 
benefits  of  which  are  generally  unknown  and  should  be  evaluated. 
Further,  these  responses  might  be  supplemented  by  other  means  for 
responding  that  are  profitable  relative  to  competing  uses  of  resources. 

Designing  and  evaluating  systems  are  the  subject  of  Sec.  V.  The 
fundamental  design  idea  that  follows  from  Secs.  II  through  IV  is  that 
management  is  a  resource  with  costs  and  benefits,  which  is  most 
likely  to  be  useful  in  situations  of  great  uncertainty.  We  present  a 
collection  of  generic  strategies  for  uncertain  situations,  mostly  to  il¬ 
lustrate  the  tradeoff  between  buying  management  and  buying  other 
things,  but  also  as  a  source  of  design  ideas.  For  system  evaluation, 
we  discuss  such  familiar  solutions  as  adding  some  statistical  variabil¬ 
ity  and  management  adaptations  to  models  or  performing  more  thor¬ 
ough  sensitivity  analyses.  We  do  not  denigrate  the  usefulness  of 
these  methods  when  they  are  appropriate,  but  we  argue  that  they  are 
sufficient  far  less  often  than  standard  treatments  suggest.  When 
they  are  insufficient,  which  we  believe  to  be  the  case  most  of  the  time, 
analyses  based  on  models  and  sensitivity  analysis  need  a  different 
kind  of  logic  to  connect  them  to  the  world.  We  discuss  two  logically 
clear  alternatives:  implementing  a  proposed  system  as  a  prototype 
and  employing  a  fortiori  arguments. 


II.  THE  COMMON  MODE  OF  POLICY  ANALYSIS 
AND  ITS  SHORTCOMINGS 


SUMMARY  OP  THE  COMMON  MODE1 

Stokey  and  Zeckhauser  (1978,  p.  5)  give  a  typical  breakdown  of 
policy  analysis  into  five  stages:  (1)  establishing  the  context,  (2)  laying 
x-  out  the  alternatives,  (3)  predicting  the  consequences,  (4)  valuing  the 

outcomes,  and  (5)  making  a  choice.  This  report  is  mostly  a  critique  of 
the  common  approach  to  the  first  four  stages,  although  it  has  impli- 
cations  for  stage  5.  Stage  2  corresponds  to  what  we  have  called  sys¬ 
tem  design,  and  stages  3  and  4  together  constitute  what  we  have 
called  evaluation.  Stage  1,  establishing  the  context,  and  stages  3  and 
4,  evaluation,  generally  use  a  stylized  representation  of  the  problem. 

The  stylized  representation  may  have  as  many  as  three  parts:  (1)  a 
tasking 2  posited  for  the  system  under  study,  (2)  a  mechanism  that 
turns  the  tasking  into  a  load  on  the  system,  and  (3)  the  managed  sys¬ 
tem  under  study,  which  is  taxed  by  the  load  and  consumes  resources 
to  service  it.  Policymakers  and  analysts  manipulate  resources  sup¬ 
plied  to  the  system  in  the  hope  of  achieving  some  desirable  outcome. 


Tasking 

A  system  cannot  be  asked  to  do  everything  that  might  need  to  be 
done  in  any  conceivable  circumstance.  It  must  be  given  specific  rc- 
sponsibilities  and  conditions  in  which  it  is  expected  to  meet  those  re¬ 
sponsibilities.  These  constitute  its  tasking 3  and  are  the  main  burden 
of  stage  1,  Setting  the  Context.  Responsibilities  are  chosen  generally 
by  the  highest-level  policymakers,  but  operators  and  analysts  within 
the  mechanism  and  the  managed  system  make  many  detailed  choices. 
Conditions  are  features  of  the  world  with  which,  in  the  common  mode 
of  analysis,  taskers  and  operators  are  told  they  must  live.  The  condi- 

lIt  is  Dot  our  object  to  do  the  definitive  survey  on  policy  analysis  or  uncertainty  but 
to  present  a  point  of  view  and  enough  references  that  an  interested  reader  can  pursue 
aspects  of  it. 

this  term  is  sometimes  used  for  very  detailed  instructions  given  to  operators.  We 
use  it  for  high-level  assignments  given  to  organizations. 

terminology  varies  tremendously  in  this  area.  Draper  et  al.  (1987)  follow  one 
common  usage  by  lumping  both  responsibilities  and  conditions  under ‘scenarios.”  We 
prefer  to  distinguish  the  two. 
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tions  are  treated  as  if  they  are  either  truly  uncontrollable  or  not 
within  the  purview  of  those  setting  the  tasking. 

In  the  context  of  the  Air  Force  example  we  are  using,  Air  Force 
planning  agencies  derive  a  detailed  tasking  from  guidance  provided 
by  higher  authorities,  to  compute  spare  parts  investments  and  repair 
requirements.  The  responsibilities  in  the  tasking  consist  of  the  war 
that  the  Air  Force  will  try  to  be  ready  to  fight:  a  single  deployment 
schedule,  flying  program,  and  support  plan.  The  chosen  war  is,  of 
course,  constrained  by  the  possibilities  allowed  by  the  current  force 
structure.  The  conditions  in  the  tasking  are  a  single  set  of  part  fail¬ 
ure  rates,  attrition  rates  for  aircraft  in  combat,  times  required  to  re¬ 
pair  and  ship  parts,  and  many  similar  things.  These  are  treated  as  if 
they  are  unchangeable  and  are  used  to  compute  spare  part  purchase 
and  ropair  requirements  as  well  as  war  reserve  material  require¬ 
ments.4  We  will  argue  later  that  some  of  these  conditions  are  in  fact 
subject  to  the  control  of  managers  and  sometimes  should  be  treated  as 
such. 

In  summary,  then,  in  the  stylized  representation  used  in  the  com¬ 
mon  mode  of  analysis,  the  tasking  is  represented  by  one  responsibility 
or  a  small  number  of  them  and  by  conditions  under  which  the  system 
will  face  those  responsibilities. 

Mechanism 

The  tasking  itself  does  not  produce  a  load  on  the  system  under 
study,  but  it  does  provide  inputs  to  a  mechanism  that  produces  the 
load.  In  the  Air  Force’s  spares  and  repair  requirement  calculations, 
the  tasking  is  turned  into  demands  on  the  repair  and  resupply  system 
by  models  that  relate  flying  hours  or  sorties  to  probability  distribu¬ 
tions  of  demands  for  replacement  parts.  In  general,  the  mechanism 
in  the  stylized  representation  is  a  model  that  takes  as  inputs  the  re¬ 
sponsibilities  and  conditions  of  the  tasking  and  produces  a  load  on  the 
system  under  study. 


4When  multiple  wartime  operational  tasking*  exiat  for  a  unit,  the  analysts  assign  it 
a  "worst  case"  support  plan,  but  they  do  not  evaluate  the  consequences  of  potential 
deviations  in  other  factors,  such  as  flying  hours  or  repair  times,  that  might  aflect 
requirements. 
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Managed  System 

Once  the  mechanism  has  produced  a  load  on  the  system,  that  sys¬ 
tem  consumes  resources  to  service  the  load.  In  fact,  the  system  does 
not  simply  draw  in  new  resources  and  consume  them,  it  also  manages 
resources  that  it  owns.  We  use  the  term  managed  system  to  empha¬ 
size  this.5 

In  the  Air  Force  example,  the  repair  and  resupply  system  removes 
broken  parts  from  planes,  replaces  them  or  orders  replacements,  and 
junks  or  repairs  the  broken  parts.  Most  models  of  this  process  treat 
the  material  flows  as  if  they  move  immutably  through  an  inert  and 
infinitely  expandable  pipe.  The  models  ignore  the  effects  on  system 
performance  of  limited  transportation  and  repair  and  of  adaptive  be¬ 
havior  within  the  system.  In  some  analyses,  the  managed  system  is 
not  explicitly  represented,  even  though  it  may  be  of  central  impor¬ 
tance.  For  example,  the  Army’s  munitions  requirements  computa¬ 
tion6  does  not  model  supply  at  all.  It  assumes  that  the  supply  system 
delivers  the  right  munitions  to  the  right  place  at  the  right  time. 

The  notions  of  tasking,  mechanism,  and  managed  system  relate 
straightforwardly  to  the  five  stages  of  policy  analysis.  Stage  1,  estab¬ 
lishing  the  context,  is  mostly  about  specifying  the  tasking.  Stage  2, 
laying  out  the  alternatives,  most  naturally  involves  the  managed  sys¬ 
tem,  although  it  may  involve  constraints  placed  on  elements  of  the 
mechanism.  Stage  3,  predicting  the  consequences,  involves  the  be¬ 
havior  of  both  the  mechanism  and  the  managed  system.  Stage  4, 
valuing  the  outcomes,  is  generally  an  evaluation  of  the  managed  sys¬ 
tem  and  of  the  adequacy  of  the  resources  supplied  to  it. 

THREE  ROLES  OF  POLICY  ANALYSIS 

Builder  (1989,  Ch.  9)  describes  several  roles  policy  analyses  may 
fill.  He  makes  particular  mention  of  three  roles:  operations  analysis, 
systems  analysis,  and  requirements  analysis.  Operations  analysis  is 
the  analysis  of  an  existing  process  or  thing  that  can  be  measured. 
Systems  analysis  is  the  comparison  of  alternative  things  or  processes 
that  are  often  mostly  hypothetical  and  thus  cannot  be  measured. 
Requirements  analysis  takes  place  after  a  systems  analysis  has  se- 

sIn  the  following  discussion,  the  term  "manager"  refers  to  any  person  in  the 
managed  system  who  can  react  effectively  to  events.  In  our  lexicon,  a  supply  clerk  who 
expedites  badly  needed  requisitions  is  as  much  a  manager  as  a  shop  chief  who  sets 
repair  priorities.  Managers  are  those  who  adapt  to  changing  circumstances  by 
reallocating  time,  effort,  material,  money,  or  information. 

*See  Girardini  (1989). 
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lected  an  alternative  and  results  in  a  specific  design  or  a  decision 
about  how  much  of  the  thing  to  buy.  In  the  Air  Force  example,  a  sys¬ 
tems  analysis  would  compare  alternative  ways  of  organizing  the 
spares  and  repair  system,  a  requirements  analysis  would  determine 
how  much  inventory  and  repair  capacity  to  buy,  and  an  operations 
analysis  would  examine  the  chosen  system  in  operation  and  adjust 
the  earlier  choices.  All  five  stages  of  a  policy  analysis  and  each  of  the 
notions  of  tasking,  mechanism,  and  managed  system  may  be  present 
in  any  of  these  kinds  of  analyses,  although  some  may  be  implicit  or 
vestigial. 

TWO  SHORTCOMINGS  OF  TBS  COMMON  MODE 

Most  analyses  proceed  as  if  some  things  are  fixed,  known  charac¬ 
teristics  of  the  world;  other  things  not  fixed  by  nature  can  be  set  by 
assumption;  and  everything  else  can  be  chosen  by  policymakers  or 
analysts  working  for  policymakers.  In  practice,  this  means  that  the 
mechanism  and  the  managed  system  are  usually  treated  as  fixed, 
known  characteristics  of  the  world.  All  too  often,  so  is  the  tasking. 
The  resources  available  to  the  managed  system  to  meet  the  predicted 
load  remain  to  be  set  by  policymakers  or  their  analysts.  Thuc,  spares 
and  repair  analysts  add  more  components  or  repair  authorizations  to 
improve  aircraft  availability  or  the  chance  that  an  item  will  be  in 
stock  under  a  single  peacetime  or  wartime  tasking.  These  analyses 
usually  attempt  to  discover  the  single  optimum  resource  mix  needed 
to  meet  the  assigned  tasking,  either  exactly  or  with  some  confidence 
level.  They  generally  make  no  certification  about  whether  that  re¬ 
source  mix  would  be  appropriate  for  other  taskings. 

This  common  approach  has  two  important  shortcomings.  First, 
large  and  often  dominant  sources  of  uncertainty  are  omitted. 
Although  it  is  traditional  to  use  probability  distributions  to  represent 
some  uncertainty,  it  is  also  traditional  to  assume  away  sources  of 
uncertainty  that  may  dwarf  those  that  are  represented.  For  example, 
the  responsibilities  and  conditions  in  the  tasking  can  rarely  be  pre¬ 
dicted  with  confidence,  models  of  the  mechanism  and  managed  sys¬ 
tem  are  not  often  validated,  and  when  they  are  validated  they  usually 
show  discouraging  inaccuracy.  All  these  facts  diminish  the  confidence 
of  the  predictions  embodied  in  the  tasking,  mechanism,  and  managed 
system  and,  if  properly  counted,  will  change  the  outcomes  of  analyses. 

The  common  approach  usually  treats  mechanism  and  managed 
system  as  if  they  are  inert,  ignoring  an  important  source  of  uncer¬ 
tainty  in  predictions  and  possibilities  for  designing  better  systems. 
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As  an  example  of  the  first  point,  models  used  to  forecast  spare  parts 
requirements  assume  that  the  behavior  of  pilots,  ground  crews,  and 
repair  managers  do  not  affect  demands.  Conversely,  they  assume 
that  the  volume  of  demands  or  the  needs  of  the  supported  force  do  not 
affect  the  performance  of  the  repair  and  supply  system.  These  models 
of  complex  human  systems  ignore  the  interconnections  among  sys¬ 
tems  and  the  effects  of  incentives  and  exigency  on  the  people  in  those 
systems.  As  for  the  second  point,  by  treating  the  managed  system  as 
inert,  analysts  may  forfeit  the  most  important  messages  in  an  analy¬ 
sis:  Managers  must  and  do  respond  to  the  unexpected,  the  system 
should  provide  “levers”  with  which  they  can  respond,  and  analysts 
must  evaluate  the  costs  and  benefits  of  such  proposed  levers  as  alter¬ 
natives  to  other  uses  of  resources. 


IIL  UNCERTAINTY  IS  PERVASIVE  BUT 
REPRESENTED  INADEQUATELY 


TWO  KINDS  OF  UNCERTAINTY 
Statistical  Uncertainty1 

Statistical  uncertainty  is  variability  observed  in  repeatable  phe¬ 
nomena.  “Observed"  means  that  one  must  have  some  measurements 
of  the  phenomena.  A  phenomenon  is  “repeatable"  if  the  conditions 
are,  or  can  be,  created  repeatedly  and  if  the  entities  subjected  to  the 
conditions  are  either  the  same  each  time,  or  are  different  but  indis¬ 
tinguishable.  (A  technical  condition  called  “exchangeability”  is  re¬ 
quired,  but  the  informal  terms  “repeatable”  and  “indistinguishable” 
are  adequate  for  our  purpose.)  This  implies  a  stability  or  persistence 
of  the  variation  through  time,  in  aggregate  terms.  When  some  phe¬ 
nomenon  is  observable  and  a  case  can  be  made  that  it  is  repeatable,  it 
is  straightforward  to  reach  consensus  on  how  to  describe  the  uncer¬ 
tainty  of  future  observations  of  it.  In  this  sense,  it  is  uncontroversial 
to  speak  of  the  existence  of  a  probability  distribution  from  which  fu¬ 
ture  observations  of  the  phenomenon  will  be  drawn. 

Taken  at  face  value,  the  notion  of  statistical  uncertainty  postulates 
that  invisible  random  number  generators  are  out  there  in  the  world 
producing  variability.  This  postulate  does  not  withstand  hard 
scrutiny:  We  use  probabilistic  models  when  we  cannot  or  choose  not 
to  model  in  greater  detail2  or  when  we  deliberately  introduce  ran¬ 
domness,  as  in  random  survey  samples.  Nonetheless,  the  idea  can  be 

'The  notion  of  ‘statistical  uncertainty,*  as  distinct  from  atate-of-the-world 
uncertainty,  goes  back  at  least  to  Frank  Knight  in  the  1920s.  The  term  "statistical 
uncertainty”  was  used  by  Fisher  (1971)  and  Quade  (1982),  but  several  other  names 
have  been  used.  Classical  statistical  theory  calls  it  "probability,”  although  nowadays 
that  term  generally  describes  other  kinds  of  uncertainty  as  well.  Steinbrunner  (1974) 
called  it  "risk,"  and  Hitch  (1960)  called  it  "insurable  risk,”  in  the  sense  that  insurance 
companies  can  study  and  treat  large  classes  of  individuals  with  given  characteristics  as 
the  bomb  tests  are  treated  in  the  text.  Madansky  (1968)  called  it  “objective 
probability,"  which  captures  the  idea  that  the  variability  is  inherent  in  the  measured 
phenomenon,  not  the  observer,  and  can  be  observed  and  perhaps  definitively 
characterized. 

2Two  canonical  examples  of  random  mechanisms  are  coin  (lips  and  subatomic 
particles.  Coin  (lips  are  deterministic  given  initial  velocity  and  revolutions  per  second 
(DcGroot,  1986).  We  find  writing  on  subatomic  particles  to  be  cither  mistaken  or 
obscure  on  the  subject  of  inherent  randomness,  although  it  is  roost  unlikely  that 
current  probabilistic  models  of  subatomic  particles  will  be  replaced  by  deterministic 
models. 
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useful,  and  it  is  an  adequate  approximation  in  parts  of  some  prob¬ 
lems.  For  example,  gravity  bombs  can  be  dropped  repeatedly  from 
planes  onto  test  ranges,  and  their  accuracy  can  be  measured.  The 
tests  can  control  for  type  of  plane,  type  of  bomb,  altitude,  speed,  and 
the  like.  If  these  conditions  are  fixed,  the  bomb's  error  (called 
"ballistic  dispersion”)  will  still  vary  and  the  notion  of  statistical 
uncertainty  will  capture  the  variation  adequately.  If  the  same  condi¬ 
tions  can  be  fixed  for  future  uses  of  the  same  type  of  bomb,  and  if  they 
are  all  of  the  conditions  that  affect  bomb  error,  then  the  notion  of  sta¬ 
tistical  uncertainty  will  be  a^nuate  to  describe  the  predictive  uncer¬ 
tainty  about  those  future  ;f  the  bomb.  If  the  conditions  cannot 

be  fixed  or  some  other  cc  -  i  can  affect  bomb  error,  then  statistical 

uncertainty  may  not  be  '■.deqaate. 

For  example,  suppose  a  bomb’s  deviation  from  its  target  is  de¬ 
scribed  by  two  components,  one  each  for  the  errors  in  the  northerly 
and  easterly  directions.  It  is  common  to  use  a  bivariate  normal 
(Gaussian)  probability  distribution  to  represent  the  variation  in  these 
two  errors.  Bivariate  normal  distributions  are  specified  by  five  pa¬ 
rameters:  the  means  and  variances  of  the  two  components,  and  their 
correlation.  Statistical  uncertainty  about  the  bomb’s  accuracy  can  be 
broken  into  two  parts:  (1)  uncertainty  about  the  probability  distribu¬ 
tion  from  which  the  ballistic  errors  are  drawn  (e.g.  uncertainty  about 
the  parameters  of  the  error  distribution),  and  (2)  uncertainty  given 
the  probability  distribution  of  the  ballistic  errors.  The  first  of  these 
uncertainties  can  be  reduced  by  taking  a  larger  sample  in  the  test 
(that  is,  parameters  can  be  estimated  with  greater  precision),  but  the 
second  one  cannot.  The  latter  is  sometimes  represented  in  military 
models  and,  less  frequently,  in  operations  research  models  generally, 
but  the  former  is  almost  never  represented. 

State  of  the  World  Uncertainty3 

Many  uncertain  things  are  either  not  repeatable,  not  observed  or 
observable,  or  both.  Hitch  (1960)  gives  the  example  of  the  behavior  of 
a  sentient  opponent  in  a  game:  In  such  instances  as  the  escalation  of 
a  superpower  conflict,  the  phenomenon  is  not  repeatable  and 
(fortunately)  not  observed.  Whatever  statements  of  uncertainty  we 
may  wish  to  make  about  such  a  situation,  they  cannot  have  the  same 

■^iaher  (1971)  used  the  term  'state-of-the-world*  uncertainty,  but  a*  with  atatiatical 
uncertainty,  others  have  used  different  name*.  Hitch  (I960)  used  'genuine 
uncertainty*  and  Steinbrunner  ( 1974)  used  'structural  uncertainty.* 
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basis  as  the  statements  we  could  make  about  bomb  accuracy  under 
specified  conditions. 

State-of-the-world  uncertainty  can  creep  into  the  problem  of  bomb 
accuracy  when  predictions  must  be  extrapolated  from  conditions  that 
have  been  tested  to  conditions  that  have  not.  Such  extrapolations  re¬ 
quire  a  model  relating  the  conditions  to  bomb  error,  and  the  accuracy 
of  the  model  can  be  a  central  concern.  In  the  Air  Force  spares  and  re¬ 
pair  example,  the  models  of  part  failures  are  used  to  extrapolate  from 
peacetime  flying  programs  to  wartime  flying  programs  that  call  for 
many  more  flying  hours  per  aircraft.  For  this  extrapolation,  the 
state-of-the-world  uncertainty  can  cover  orders  of  magnitude.4 

Behavior  of  people  in  the  mechanism  or  the  managed  system  adds 
a  more  complex  layer  to  the  state-of-the-world  uncertainty.  In  World 
War  II,  aircrews  were  not  generally  reassigned  to  targets  that  they 
had  missed  and  so  lacked  an  incentive  to  follow  the  hazardous  though 
effective  bombing  practices  used  in  testa  of  bomb  accuracy.  Instead, 
the  incentives  were  such  as  to  incline  them  to  safer  but  less  effective 
techniques,  which  would  diminish  actual  bomb  accuracy.6  For  an¬ 
other  example,  field  tests  of  Air  Force  units  show  that  rates  of  part 
failures  depend  on  the  incentives  acting  on  the  pilots:  If  bombing  ac¬ 
curacy  is  being  measured,  pilots  have  an  incentive  to  report  failures 
in  their  planes’  fire  control  systems;  but  if  accuracy  is  not  measured, 
they  have  no  such  incentive,  and  reported  failures  drop.6  Human  in¬ 
terventions,  even  by  people  “on  our  side,"  introduce  uncertainty  that 
is  often  difficult  or  impossible  to  characterize. 

The  behavior  of  the  people  outside  the  managed  system  introduces 
still  another  layer  of  uncertainty.  This  state-of-the-world  uncertainty 
has  two  variants.  First,  actions  of  those  outside  the  managed  system 
may  be  unpredictable.  For  example,  before  Pearl  Harbor,  many 
American  military  analysts  believed  that  aerial  torpedoes  would  be 
ineffective  there,  because  they  would  probably  hit  bottom  when 
dropped  in  such  a  shallow  harbor.  However,  Japanese  aviators  devel¬ 
oped  a  special  fin  that  overcame  the  problem  (Wohlstetter,  1962). 

The  interconnection  of  behavior  within  the  system  with  behavior 
outside  it  may  induce  sequences  of  changes  on  both  sides  that  are  im- 

4Donaldson  and  Sweetlaod  (1968),  Embry  (1984),  Pipp  (1988),  and  unpublished 
RAND  research  by  G.C.  Crawford  and  M.  Kamins  and  by  M.  Berman  and  T.  Lippiatt 
all  suggest  that  demands  per  flying  hour  could  decrease  by  as  much  as  a  half  in 
wartime  for  some  important  and  expensive  parts.  These  pieces  of  evidence  are  the 
subjects  of  a  heated  diapute,  but  it  is  clear  that  nobody  has  a  grasp  of  whst  would 
happen. 

^Personal  communication  from  J.  Stock  flsch. 

•Personal  communication  from  H.  Shulman. 
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possible  to  predict.  For  example,  when  the  British  installed  centime¬ 
ter  radar  on  their  anti-submarine  planes,  German  submariners  re¬ 
acted  by  changing  their  tactics  and  diving  when  they  sighted  a  plane. 
The  British  eventually  deduced  the  depth  the  U-boats  would  reach  by 
the  time  the  plane  reached  them  and  began  to  set  their  depth  charges 
accordingly,  with  devastating  results.  The  Germans  responded  with 
the  snorkel,  which  reduced  the  radar  signature  of  their  submarines. 
Even  after  the  centimeter  radar  had  been  deployed,  it  would  have 
been  impossible  to  predict  the  outcome  of  this  cycle  of  action  and  re¬ 
action.7 

One  implication  of  these  examples  is  that  analysts  may  not  even  be 
able  to  put  bounds  on  plausible  outcomes.  Such  bounds  are  essential 
to  sensitivity  analyses  as  commonly  used;  and  without  such  bounds, 
the  logic  of  sensitivity  analysis  must  change.  We  discuss  this  further 
in  Sec.  V. 

Such  tools  as  subjective  probability  have  been  used  to  represent 
this  kind  of  uncertainty,  because  subjective  probability  is  manipu¬ 
lated  according  to  the  same  rules  as  statistical  uncertainty.8  (Many 
proponents  of  subjective  probability  argue  that  statistical  uncertainty 
is  a  special  case  of  subjective  probability.)  Sometimes,  state-of-the- 
world  uncertainty  extends  to  uncertainty  about  what  the  possible 
outcomes  are,  as  in  the  centimeter  radar  case;  and  in  these  cases 
subjective  probability  may  not  be  useful.9  However,  even  leaving  this 
objection  aside,  many  who  are  not  hostile  to  the  idea  of  subjective 
probability  do  not  accept  its  use  for  describing  state-of  the-world 
uncertainty.  They  argue  that  a  subjective  probability  distribution  is 

7Emery  and  Tri»t  (1965)  discussed  four  levels  of  complexity  of  interconnection 
between  organizations  and  their  environments.  This  paragraph  is  close  to  the 
environment  they  call  a  “turbulent  Held.’ 

4 Hodges  (1987)  surveyed  the  statistical  literature  on  methods  for  handling 
uncertainty  about  model  extrapolations.  Methods  involving  subjective  probability  to 
greater  or  lesser  extents  were  found  to  be  the  only  formal  methods  that  had  been 
considered.  Draper  et  al.  (1987),  using  different  terminology,  surveyed  the  literature  of 
formal  methods  for  handling  all  the  varieties  of  state-of-the-world  uncertainty  and 
made  a  similar  finding.  Alternatives  to  probability  (e.g.,  fuzzy  set  theory,  belief 
(unctions)  were  found  to  ofTer  no  solution  to  problems  of  extrapolation  or  specification 
and,  thus,  no  advantage  over  subjective  probability  for  this  problem. 

*Conrath  (1967)  distinguishes  four  degrees  of  uncertain  situations:  (1)  a  future 
outcome  of  interest  is  known;  (2)  the  possible  outcomes  are  known  and  a  probability 
distribution  across  them  is  known,  or,  in  subjectivist  terms,  is  agreed  upon;  (3)  the 
possible  outcomes  are  known  but  a  probability  distribution  across  them  is  not  known  or 
agreed  upon;  and  (4)  the  possible  outcomes  are  not  all  known.  The  former  three  arc 
state*  of  “bounded  uncertainty,"  and  the  last  is  a  state  of  “unbounded  uncertainty." 
Conrath  applies  these  four  degrees  of  uncertainty  to  all  three  parts  of  the  classical 
decision  problem:  state  of  nature,  action,  and  value  (utility,  loss). 
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an  assertion  of  information  like  any  other  assertion;  and  if  it  is  not 
founded  on  observations,  on  the  basis  of  which  other  people  might  be 
persuaded  to  accept  it,  then  it  is  useless  and  a  distraction.  For  the 
most  part,  we  adopt  this  view  here. 

EXAMPLES  AND  CRITIQUE  OF  THE  TREATMENT 
OF  UNCERTAINTY 

In  our  discussion  of  policy  analysis,  we  described  three  roles  of 
analysis:  operations  analysis,  systems  analysis,  and  requirements 
analysis.  The  importance  of  statistical  and  state-of-the-world  uncer¬ 
tainty  varies  depending  on  the  role  of  analysis.  In  operations 
analysis,  the  system  is  operating  and  can  be  measured,  so  much  of 
relevant  uncertainty  will  be  statistical.  In  systems  analysis,  the 
alternatives  are  largely  hypothetical,  so  measurements  are  not 
available.  In  addition,  many  important  aspects  of  the  conditions  and 
responsibilities  will  be  hypothetical.  Most,  perhaps  all,  of  the 
relevant  uncertainty  will  be  state-of-the-world.  Requirements 
analysis  falls  between  the  other  two  roles.  In  the  Air  Force  spares 
and  repair  example,  the  system  is  operational,  but  it  must  be 
outfitted  against  the  possibility  of  a  war.  Thus,  both  kinds  of 

'•‘’.-tainty  come  into  play. 

Lxaruple  3:  Air  Force  Spares  end  Repair  Requirements 

The  responsibilities  in  the  tasking  are  derived  from  guidance  pub¬ 
lished  by  the  Department  of  Defense  and  the  services  each  year.  This 
guidance  contains  information  about  several  possible  contingencies 
for  each  year  of  a  five-year  plan  and  directs  which  contingency  is  to  be 
used  to  compute  requirements.  The  Air  Force  produces  a  series  of 
war  mobilization  planning  documents  based  on  the  guidance,  contain¬ 
ing  flying  hours,  sorties,  force  structures,  projected  attrition  rates, 
deployment  schedules,  and  other  information  for  each  squadron  in  the 
chosen  contingency.  This  information  is  combined  with  detailed  air¬ 
craft  configuration  dote  to  estimate  the  tasking  (flying  hours  per  unit 
of  time)  that  each  aircraft  part  will  experience.  A  similar  tasking  is 
derived  for  peacetime  requirements,  based  on  the  forces’  training 
needs. 

In  most  Air  Force  requirement  models,  removals  of  parts  are  mod¬ 
eled  as  Poisson  random  variables  with  means  proportional  to  flying 
hours,  and  the  demand  rate  that  specifies  this  proportion  is  derived 
from  peacetime  failure  data.  In  some  requirement  models,  the 
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Poisson  model  is  replaced  by  a  negative  binomial  model  for  which  the 
variance-to-mean  ratio  is  set  in  several  ways. 

Repair  and  transportation  times  are  the  most  important  elements 
in  requirement  models  of  the  managed  system.  Usually  these  times 
are  assumed  to  be  unvarying  or  are  exponential  random  variables. 
Management  is  assumed  away  by  postulating  ample  (actually  infi¬ 
nite)  repair  and  transportation  capacity. 

Critique  of  Example  1 

Requirements  for  spares  and  repair  are  computed  using  one  ap¬ 
proved  contingency.  This  means  that  war  mobilization  planning  doc¬ 
uments  do  not  address  any  of  the  state-of-the-world  uncertainty  as¬ 
sociated  with  how  the  enemy  might  prosecute  the  war,  with  how  U.S. 
commanders  might  design  flying  programs  in  response,  with  the  ef¬ 
fects  of  battle  damage  to  bases,  or  with  difficulties  in  operating  the 
support  system.  Instead,  resources  are  procured  for  essentially  one 
possible  war,  in  a  fairly  benign  environment 

To  illustrate  the  omission  of  these  uncertain  features  of  the 
problem,  the  effect  of  omitting  battle  damage  from  evaluations  of 
parts  stocks  and  alternative  repair  and  resupply  systems  is  being 
considered  in  ongoing  but  unpublished  RAND  research.  An  analysis 
of  three  airbases  subjected  each  base  to  a  typical  enemy  attack  on  the 
first  day  of  the  “war,”  and  varying  amounts  of  spare  parts  and  repair 
capability  were  lost.  The  expected  number  of  aircraft  that  would  not 
be  fully  mission  capable  thirty  days  later  was  roughly  twice  the  num¬ 
ber  projected  in  a  tasking  that  omitted  base  damage  but  was  other¬ 
wise  the  same.  Plainly  base  damage  is  important  and  its  extent  is 
uncertain.  If  included  it  will  affect  the  requirement,  but  it  is  ignored. 

One  obvious  alternative  to  the  single  tasking  is  a  group  of  taskings. 
Some  readers  have  argued  that  a  central  purpose  of  the  single  war 
used  in  a  requirement  computation  is  to  provide  a  “level  playing  field” 
for  the  various  budgetary  contenders,  and  to  ensure  that  the  compu¬ 
tation  produces  a  single  number,  not  a  range  of  numbers.  But  a  spec¬ 
ified  group  of  taskings  can  provide  as  level  a  playing  field  as  a  single 
tasking  and  need  not  force  the  single  requirement  number  to  be  re¬ 
placed  by  a  range  of  numbers.  Instead,  the  single  number  could  be 
tested  against  the  range  of  taskings  to  see  if  it  would  suffice  for  all. 

Some  uncertainty  does  appear  in  the  model  of  the  spares  and  repair 
mechanism.  It  is  all  treated  as  if  it  were  statistical  uncertainty, 
captured  in  the  Poisson  or  negative  binomial  model  for  part  failures. 
This  model  posits  several  specific  things,  including  constant  average 
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rate  of  demands  per  flying  hour  or  sortie,  with  the  same  rate  prevail¬ 
ing  in  peacetime  and  wartime.  Even  taking  this  model  as  accurate  for 
the  moment,  the  standard  approach  ignores  one  important  aspect  of 
statistical  uncertainty — the  uncertainty  about  the  demand  rates  in¬ 
serted  in  the  model.  However,  the  model  is  not  accurate  for  many 
parts,10  so  its  use  for  extrapolation  suppresses  uncertainty  about 
predictions  of  failures  wartime.  In  sum,  the  mechanism  omits 
large  and  important  state-of-the-world  uncertainty,  and  its  represen¬ 
tation  of  statistical  uncertainty  has  little  to  recommend  it. 

The  same  is  true  for  the  managed  system.  Both  constraints  and 
management  are  eliminated  from  the  model  of  repair  and  manage¬ 
ment,  when  in  fact  both  exist.  The  Department  of  Defense  and  the 
services  have  formal  systems  to  ensure  that  priority  is  given  to  those 
needs  that  most  affect  the  forces’  ability  to  accomplish  their  missions. 
Although  studies  of  priority  rules  show  that  their  effect  can  be  dra¬ 
matic,  these  managerial  adaptations  and  others  that  are  known  to  ex¬ 
ist  are  ignored  in  the  requirements  process. 


Example  2:  Army  Munitions  Requirements11 

As  with  the  Air  Force  spares  and  repair  computation,  the  Army’s 
munitions  requirement  calculation  uses  a  single  possible  war.  In  ad¬ 
dition,  only  one  of  the  suite  of  models  used  has  stochastic  elements 
and  is  run  repeatedly  to  generate  variation  in  ammunition  expendi¬ 
ture  rates.  Also,  the  munitions  requirement  uses  the  average  of  these 
rates  and  ignores  variation.  Moreover,  data  used  as  inputs  to  these 
models,  notably  kill  probabilities,  are  themselves  uncertain,  some¬ 
times  very  much  so,  but  this  is  ignored.  Finally,  the  models  of  war¬ 
fare  themselves  are  largely  conjectural,  because  empirical  study  to 
build  scientifically  valid  models  has  not  and  probably  can  not  be  done. 
The  Army’s  analysts  attempt  to  account  for  these  difficulties  by  using 
conservative  values  of  input  data,  but  it  is  difficult  to  see  how  conser¬ 
vatism  can  be  assessed  in  the  face  of  such  massive  uncertainty. 


10Thirty-five  years  of  data  consistently  contradict  the  Poisson  assumption  for  high- 
failure  items.  Mean  demands  are  not  constant  even  in  peacetime,  are  not  proportional 
to  Hying  hours,  and  probably  cannot  be  projected  to  wartime  by  a  linear  extrapolation. 
Evidence  against  the  Poisson  assumption  for  high-<iemand  items  is  contained  in  Brown 
and  Geisler  (1964);  Youngs,  Geisler,  and  Brown  (1956);  Brown  (1966);  Astrakhan, 
Brown,  and  Houghton  (1961);  Astrachan  and  Sherbrooke  (1964);  Hodges  (1986);  and 
Crawford  (1988),  among  others.  Evidence  against  constant  mean  demand  rates  and 
proportionality  to  flying  hours  is  in  Astrachan,  Brown,  and  Houghton  (1961),  Hodges 
( 1986),  and  Crawford  ( 1988). 

11  See  Girardini  (1989). 
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Example  3:  Air  Force  Munitions  Requirements12 

As  with  the  Air  Force’s  spares  and  repair  requirement,  a  single 
contingency  is  used,  along  with  official  rates  of  aircraft  attrition  and 
similar  model  inputs.  In  contrast  to  the  Army’s  calculation,  some  but 
not  all  determinations  of  munitions  effectiveness  account  for  statisti¬ 
cal  uncertainty.  The  discussion  of  uncertainty  concludes  with  the 
judgment  that  adding  more  of  these  sources  of  statistical  uncertainty 
“would  add  little  or  nothing  to  the  plausibility  of  the  results  of  a 
model  operating  within  this  framework”13  (i.e.,  when  so  muoh  state- 
of-the-world  uncertainty  is  ignored). 

Nonmilitary  Examples 

We  cite  some  nonmilitary  examples  here  to  illustrate  the  wide- 
ranging  nature  of  the  deficiency  in  the  treatment  of  uncertainty. 
Policy  about  low-level  radiation  hazard  is  made  by  administering  high 
doses  of  radiation  to  lab  animals,  counting  the  resulting  cancers,  and 
extrapolating  to  low  levels  of  radiation  and  to  humans.  Both  parts  of 
this  extrapolation  are  highly  uncertain,  and  that  uncertainty  is  rou¬ 
tinely  ignored.14  Quantitative  risk  assessment  generally  is  plagued 
by  such  problems.16  Fiscal  policy  is  made  using  macroeconomic 
models,  which  are  subject  to  great  dispute  and  require  projections  of 
conditions  that  are  conjectural  at  best.16  Psychological  and  educa¬ 
tional  research  make  extensive  use  of  so-called  structural  models, 
which  rely  on  assumptions  that  are  inherently  un verifiable. 17  Even 
the  determination  of  physical  constants — as  pristine  as  possible,  one 
might  think — is  subject  to  uncertain  nonexperimental  error. 
International  standards  organizations  are  only  beginning  to  regulate 
the  reporting  of  such  errors. 16 

12See  Crawford  (1989). 

13Ibid.,  p.  42. 

MKalbfleisch  and  Prentice  (1980);  Hattii  and  Smith  (1985);  Freedman  and  Zeisel 
(1988). 

>sHattis  and  Smith  (1985);  Hattia  and  Kennedy  (1986);  Kamina  (1875);  Speed  (1985), 
among  many  others. 

I6Many  examples  are  available;  Greider's  (1981)  candid  interviews  with  David 
Stockman  are  a  good  one. 

r/The  Summer  1987  issue  of  the  Journal  of  Educational  Statistics  (Vol.  12,  No.  2)  is 
devoted  entirely  to  a  critique  of  structural  models  by  Freedman  and  to  a  discussion  by 
defenders  and  other  critics  of  such  models.  Holland  (1988)  shows  the  reliance  on 
inherently  unverifiable  assumptions. 

'®Corait6  International  de  Poida  et  Matures  ( 1981). 


IV.  MECHANISM  AND  MANAGED  SYSTEM  ADAPT 
TO  THE  UNEXPECTED 


If  the  common  mode  of  analysis  treats  uncertainty  so  poorly,  why 
aren't  more  military  systems  hopelessly  bogged  down?  One  reason  is 
that  managed  systems  and  mechanisms  a  not  inert;  they  are  sys¬ 
tems  that  adapt  to  the  unexpected.  Instead  of  ignoring  these  facts  or 
simply  treating  them  as  deficiencies  in  our  models,  we  can  use  them 
as  indications  of  how  analysts  can  help  managers  cope  with  uncer¬ 
tainty  by  helping  them  adapt,  instead  of  by  helping  them  try  to  pre¬ 
dict  the  unpredictable. 

ADAPTABILITY  OF  THE  MECHANISM 
AND  MANAGED  SYSTEM 

The  mechanism  and  managed  system  generally  involve  large  hu¬ 
man  organizations.  The  people  in  them  do  not  simply  follow  rules 
blindly,  but  respond  in  varying  ways  to  events  and  to  the  incentives 
with  which  they  are  surrounded.1  Although  this  fact  may  just  make 
it  more  difficult  to  predict  outcomes,  if  fully  appreciated  it  may  also 
reveal  opportunities  for  designing  adaptive  systems. 

To  return  to  the  Air  Force  example,  some  argue  that  the  Air  Force’s 
model  of  part  failures  is  deficient  because  it  treats  the  mechanism  as 
inert  when  it  is  not.  Some  types  of  equipment  degrade  instead  of  fail¬ 
ing  outright;  because  assessment  of  degradation  requires  human 
judgment,  it  can  be  highly  subjective.  Pilots  and  mechanics  vary  in 
the  skill  with  which  they  diagnose  faults.  Because  a  pilot  did  not  ex¬ 
ercise  some  aircraft  subsystem  in  his  last  mi  >ion,  he  may  not  observe 
that  it  has  failed.  Managers  can  change  failure  rates  with  decisions 
about  what  evidence  should  trigger  a  removal,  and  it  is  not  unusual 
for  a  single  failure  to  prompt  wholesale  inspections  and  removals. 

This,  too,  is  not  just  a  problem  for  military  analysts.  For  example, 
the  Rational  Expectations  school  of  macroeconomic  thinking  has  cast 
doubt  on  decades  of  models  and  derived  policy  with  the  hypothesis 
that  agents  in  the  economy  anticipate  the  effects  of  government 

’Hoenack  (1983)  gives  a  rich,  if  somewhat  abstract,  microeconomic  construction  of 
incentives  created  within  organizations  and  the  behavior  they  can  be  expected  to 
induce. 
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macroeconomic  policies  and  negate  those  effects  by  reacting  to  their 
expectations.2 

Air  Force  spares  and  repair  managers  respond  to  the  unexpected 
with  a  variety  of  adaptations.  For  example,  they  take  parts  from  war 
reserve  kits,  remove  parts  from  disabled  planes  to  use  in  other  planes, 
and  schedule  overtime  work,  to  name  a  few.  Similarly,  Navy  ships 
engage  in  “local  barter,”  trading  movies  for  coffee,  for  example.  More 
formally,  the  services  constantly  reallocate  material  across  unit3  to 
maintain  readiness:  In  December  1988  alone,  the  United  States  Air 
Forces  in  Europe  (USAFE)  responded  to  over  1500  redistribution 
requests  for  material  needed  by  other  USAF  units  worldwide.3 

These  adaptations  are  clearly  useful,  but  they  are  rarely  modeled 
and  even  more  rarely  evaluated  in  terms  of  their  costs  and  contribu¬ 
tions  to  overall  system  effectiveness.  Instead,  the  Air  Force’s  models 
postulate  a  system  where  management  is  unnecessary  and  has  no 
value. 

Again,  this  is  not  simply  a  phenomenon  in  military  analyses.  In  a 
study  of  the  New  York  City  Fire  Department,  analysts  found  that 
when  the  rate  of  alarms  got  high,  dispatchers  adapted  by  dispatching 
fewer  fire  engines  than  their  rules  required,  thereby  keeping  some 
engines  in  reserve.4  The  dispatchers  took  the  chance  that  some  fires 
would  not  initially  receive  enough  units,  but  they  retained  some  ca¬ 
pacity  to  respond  to  new  fires  and  to  uner.pected  severity  of  existing 
fires. 

GIVING  MANAGERS  MORE  WAYS  TO  ADAPT 

Rather  than  be  chagrined  that  we  do  not  model  managers  well, 
perhaps  we  should  note  that  they  are  doing  what  prominent  writers 
on  systems  analysis  recommend,  and  help  them  do  more  of  it.  For  ex¬ 
ample,  Hitch  (1960)  noted  that  state-of-the- world  uncertainties  are 
omnipresent  and  usually  irreducible  and  argued  that  the  appropriate 
analytical  tack  is  a  shift  “from  a  search  for  a  belter  decision  rule  to  a 
search  for  a  better  system  l,  firom  sophistication  in  judgment  to  inge¬ 
nuity  in  design  [p.  5].  .  .  .  Most  of  our  relations  are  so  un predictable 
that  we  do  well  to  get  the  right  sign  and  order  of  magnitude  of  first 
differentials.  In  most  of  our  attempted  optimizations  we  are  kidding 
our  customers  or  ourselves  or  both. ...  It  is  much  easier  to  find  a  sys- 

^See,  e.g.,  Begg  (1982). 

3Letter,  Brigadier  General  Philip  Metzler,  USAFE  DCS/Logistics,  u>  Mr.  John  Abell, 
February  1989. 

4RAND  Fire  Project  (1979),  pp.  43M32,  p.  625,  and  sec.  11.3.3 
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tem  or  strategy  that  dominates  or  nearly  dominates  some  other  sys¬ 
tem — say  the  one  currently  planned — than  it  is  to  find  a  system  that 
dominates  all  other  systems. .  .  .  And  if  we  can’t  find  such  a  system, 
we  can  frequently  invent  one.  In  fact,  it  has  been  our  experience  that 
ingenuity  is  frequently  more  profitably  exercised  in  invention  than  in 
mere  judging’’  (p.  9).5 

Analysts  involved  in  selecting  and  evaluating  the  first  geologic 
high-level  radioactive  waste  repository,  for  example,  spend  tremen¬ 
dous  effort  to  compute  a  probability  that  the  proposed  repository  will 
not  fail  for  tens  of  thousands  of  years.6  These  brave  people  attempt 
the  impossible  because  their  legislative  and  executive  masters  ask 
them  to.  The  more  pertinent  problem  is  how  the  managers  of  the 
repository  will  respond  to  breaches,  or  how  the  design  or  site  selection 
might  be  modified  to  make  it  easier  to  detect  a  failure  and  respond  to 
it  or,  indeed,  whether  the  time  spent  computing  an  uncomputable 
probability  would  be  better  spent  on  designing  management  re¬ 
sponses. 

Some  readers  have  argued  that  management  is  only  pertinent  in 
operations  analysis,  not  requirements  or  systems  analysis.  We  do  not 
agree.  For  example,  the  ability  of  managers  to  adapt  is  (or  should  be) 
at  the  heart  of  systems  analyses  of  alternative  high-level  radioactive 
waste  depositories.  Similarly,  the  efficacy  of  management  is  the  cen¬ 
tral  issue  in  our  discussion  of  the  Air  Force’s  spares  and  repair  re¬ 
quirement.  The  next  section  elaborates  this  point. 

THE  EFFICACY  OF  MANAGEMENT:  THE  CLOUT  STUDY 

A  study  of  a  collection  of  Air  Force  spare  parts  initiatives  called 
CLOUT7  included  both  operational  analysis — examining  how  repairs 
should  be  scheduled — and  requirements  analysis-— computing  quar¬ 
terly  and  annual  requirements  for  spares  and  repair.  It  added  to  the 
standard  analysis  both  more  uncertainty  and  more  management 

eIn  a  similar  vein,  AckofT  (1964)  presented  six  case  studies  of  industrial  operations 
research  problems  and  drew  the  generalization  that  ’greater  improvement  of 
performance  than  is  obtainsble  by  optimization  may  be  obtained  by  changing  the 
problem . . .  what  this  can  almost  always  do  id  make  the  system  m on  adaptive:  that  is, 
make  it  more  sensitive  and  responsive  to  changing  conditions  which  alTect  its 
performance  [p.  €]....  My  hope  is  that  we  do  not  allow  well  formulated  models  and 
methods  of  solving  them  to  give  us  a  mental  set  which  prevents  our  exploring  more 
funda.'nentrl  changes  in  the  system  than  these  models  consider*  (p.  11). 

®For  example,  Eisenberg  (1988),  Doctor  ( 1988),  Booker  (1988). 

'CLOUT  is  an  acronym  for  Coupling  Logistics  to  Operations  to  meet  Uncertainty 
and  the  Threat. 
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adaptations.  Two  examples  of  the  additional  uncertainty  are  a  nega¬ 
tive  binomial  model  for  part  failures  (instead  of  Poisson)  and  uncer¬ 
tain  airbase  damage  arising  from  enemy  attacks.  The  management 
adaptations  included  faster  depot  repair  for  critical  parts  (“priority 
repair")  and  allowing  air  bases  to  use  the  part  stocks  and  repair  ca¬ 
pacity  of  nearby  airbases  (“lateral  resupply  and  repair"),  with  all 
these  choices  made  to  maximize  the  number  of  aircraft  available  to  fly 
missions.  Both  of  these  practices  occur  now  to  varying  extents. 

The  effect  of  adding  uncertainty  to  the  standard  analysis  without 
adding  the  management  adaptations  was  to  double  the  expected 
number  of  .aircraft  that  could  not  execute  their  missions  one  month 
into  the  war.  When  the  management  adaptations  were  added,  the 
number  of  downed  aircraft  was  reduced  to  slightly  more  than  that  ob¬ 
tained  in  the  standard  analysis. 

For  operations,  one  implication  is  that  responsive  scheduling  of  re¬ 
pairs  can  have  a  dramatic  effect  on  aircraft  availability  in  uncertain 
environments.  A  system  to  schedule  depot  repairs  responsively  has 
been  developed  and  a  prototype  is  now  operating  at  the  Ogden  Air 
Logistics  Center.  Experience  is  showing  that  the  gains  projected  in 
the  CLOUT  analyses  are  real.® 

The  implication  for  requirements  calculation  is  that  the  computa¬ 
tion  is  strongly  affected  by  the  inclusion  of  management.  If  the  stan¬ 
dard  computation  were  changed  by  adding  the  sources  of  uncertainty 
considered  in  CLOUT,  then  the  computed  aircraft  availability  would 
be  diminished.  Availability  could  be  brought  back  to  more  desirable 
levels  by  buying  more  spare  parts,  by  managing  the  in  ventory  of  parts 
differently,  or  by  some  combination  of  these  two.  Each  choice  has 
costs  and  benefits  that  need  to  be  evaluated  as  part  of  the  require¬ 
ment  computation. 

The  problems  of  design  and  evaluation  in  the  CLOUT  study  are 
examples  of  considerations  generic  to  analyses  that  take  realistic  ac¬ 
count  of  uncertainty.  The  next  section  describes  in  some  generality 
these  implications  about  the  way  analysts  do  business. 


®Permonal  communication,  John  B.  Abell. 


V.  IMPLICATIONS  FOR  ANALYSTS:  DESIGN  AND 
EVALUATION  OF  SYSTEMS 


As  mentioned  in  Sec.  II,  the  five  stages  of  a  policy  analysis  are 
establishing  the  context,  laying  out  the  alternatives,  predicting  the 
consequences,  valuing  the  outcomes,  and  making  a  choice.  In  the 
terms  we  have  discussed,  stage  2  is  about  system  design,  stages  3  and 
4  are  about  system  evaluation,  and  stage  1  is  essentially  setting  the 
tasking.  A  proper  recognition  of  uncertainty  will  affect  all  of  these. 


IMPLICATIONS  FOR  SYSTEM  DESIGN:  GENERIC 
STRATEGIES  FOR  UNCERTAIN  SITUATIONS1 

One  message  of  the  last  section  is  that  management  is  a  resource 
with  costs  and  benefits  and  is  purchased  in  competition  with  other 
resources.  This  subsection  illustrates  some  possibilities  for  system 
design  and  emphasizes  the  tradeoff  between  management  and  other 
resources  by  describing  a  collection  of  generic  strategies  for  uncertain 
situations. 

Passive  and  Active  Strategies 

Strategies  for  uncertain  situations  can  be  broken  into  passive  and 
active  groups.  Passive  strategies  rely  on  size,  scale,  or  diversity: 
Passive  systems  are  ample ;  they  absorb.  Active  strategies  work  on 
the  premise  that  the  future  will  consist  of  outcomes  following  from 
specific  key  events;  the  system  must  recognize  those  key  events  and 
adapt  in  a  timely  fashion  with  some  activity  relevant  to  the  impend¬ 
ing  outcomes.  Obviously,  active  strategies  need  resources  that  are 
ample  in  some  sense,  but  they  differ  from  passive  strategies  by  adapt¬ 
ing  to  do  more  with  less.  In  this  sense,  passive  strategies  emphasize 
goods,  and  active  strategies  emphasize  management  of  goods. 

Passive  strategies  tend  to  be  advantageous  in  stable  situations 
where  the  uncertainty  that  is  present  can  be  well  understood  if  not 
diminished,  while  active  strategies  tend  to  be  advantageous  in  situa¬ 
tions  of  great  uncertainty  where  the  likely  error  of  predictions  is 
either  large  or  is  itself  quite  uncertain. 

’Madansky  (1968)  discussed  some  of  the  strategies  bated  here. 
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This  discussion  will  now  be  elaborated  with  examples  of  three  pas¬ 
sive  strategies  and  four  active  strategies. 

Examples  of  Passive  Strategies 

Buy  It  Out.  This  strategy  has  many  variants.  First,  one  can  cover 
all  the  possibilities.  For  example,  the  Maytag  repairman  doesn’t 
know  what  tools  he  will  need  at  a  given  house,  but  he  doesn’t  need  to 
because  he  carries  them  all  in  his  truck.  Second,  it  is  possible  to 
squeeze  more  out  of  what  is  already  available.  Scheduling  overtime 
in  repair  shops  i  3  one  example  of  this.  Third,  it  may  be  possible  to 
bank  resources:  Squirrels  save  nuts  for  winter,  and  people  save 
money  for  retirement.  In  all  these  examples,  the  strategy  is  to  have 
enough  to  cover  whatever  happens. 

Operate  on  a  Larger  Scale.  An  insurance  company  guards 
against  losses  from  a  single  person’s  claims  by  grouping  that  person 
with  many  similar  people.  Because  the  group  is  less  erratic  than  the 
individuals,  no  individual  needs  to  pay  enough  money  to  cover  the 
largest  insured  claim.  Auto  repair  chains  in  large  cities  use  a  similar 
strategy  when  they  have  a  central  warehouse  for  most  of  their  parts 
and  dispatch  parts  from  the  warehouse  as  needed,  instead  of  trying  to 
buy  enough  parts  to  stock  each  repair  shop  against  all  eventualities. 

Diversify.  An  insurance  company  holding  many  fire  insurance 
policies  in  one  area  is  subject  to  sudden  bankruptcy  if  all  the  policy¬ 
holders  lose  their  homes  to  a  large  fire.  The  company  guards  against 
this  by  reinsurance.  Investors  do  the  same  thing  when  they  diversify 
their  portfolios. 

Examples  of  Active  Strategies 

Get  More  Information.  One  variant  on  this  strategy  is  to  pro¬ 
crastinate  and  react  as  late  as  possible.  Procrastination  preserves  op¬ 
tions,  although  it  can  incur  costs.  Quarterbacks  use  this  strategy 
when  they  call  audible  plays  at  the  line  of  scrimmage.  A  similar 
strategy  is  practiced  in  the  Services’  depot  repair  organizations: 
While  they  develop  quarterly  repair  plans  to  meet  the  force’s  pro¬ 
jected  needs,  they  adjust  those  plans  somewhat  as  the  force’s  actual 
needs  become  known.  For  example,  if  demands  for  a  part  are  so  great 
that  wartime  capability  is  threatened,  the  depots  rush  unplanned  re¬ 
pairs  through  the  shops.  Another  variant  of  this  strategy  is  to  buy 
more  information  without  necessarily  procrastinating  first.  Providing 
current  information  about  part  inventories  to  a  responsive  depot  re- 
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pair  system  is  an  example.  Another  was  suggested  in  the  example  of 
the  repository  for  high-level  radioactive  waste:  Aggressively  monitor 
the  performance  of  the  containment  structure,  and  respond  to  it. 

Use  General  or  Flexible  Resources.  One  particularly  effective 
way  to  exploit  information  relies  on  obtaining  and  maintaining  flexi¬ 
ble  general-purpose  resources.  For  example,  the  effectiveness  of  the 
quarterback’s  audible  call  depends  on  the  ability  of  the  other  team 
members  to  memorize  and  execute  a  large  number  of  plays.  The 
CLOUT  options,  by  contrast,  are  of  little  use  to  a  repair  shop  that  can 
repair  only  one  item.  In  munitions  procurement,  flexible  manufactur¬ 
ing  provides  a  capacity  to  produce  several  different  munitions  depend¬ 
ing  on  what  the  unfolding  conflict  demands.  Aircraft  leasing  firms 
have  a  general  resource  in  their  planes.  Although  an  airline  trying  to 
lease  a  single  superfluous  plane  has  just  one  plane,  and  information 
is  useful  to  it  only  to  the  extent  that  it  helps  the  airline  lease  that 
single  plane,  a  leasing  firm  has  several  planes,  and  its  information  is 
relevant  to  its  whole  fleet.  Flexibility  can  also  be  obtained  by  selec¬ 
tion.  In  World  War  II,  the  U.S.  Army  trained  new  troops  into  the 
branches  based  on  the  requests  from  the  fielded  units.  It  did  not 
simply  train  recruits  according  to  preset  schedules  or  force  structures, 
it  adjusted  training  as  the  units  began  to  understand  their  needs. 

Push  the  Risk  onto  Someone  Else.  Airlines  face  an  uncertain 
world  under  deregulation  and  push  some  of  the  resulting  risk  onto 
leasing  firms.  The  leasing  firms  own  the  planes  and  worry  about  who 
will  use  them,  not  the  airlines.  Just-in-time  inventory  systems  push 
the  risk  of  supply  onto  the  supplier.  Wholesalers  and  job  shops  accept 
such  risks  by  adopting  one  or  more  of  the  strategies  described  earlier. 
In  general,  the  person  or  organization  accepting  such  risk  must  have 
some  advantage  if  this  strategy  is  to  work.  The  leasing  firm’s  advan¬ 
tages  are  its  flexible  resources — aircraft  that  can  be  used  anywhere — 
and  its  market  information.  The  wholesaler’s  advantages  are  its  scale 
and  its  knowledge  of  recent  or  pending  demands.  The  job  shop’s  ad¬ 
vantages  are  its  flexible  resources. 

Combine  Strategies.  Munitions  manufacturing  for  wartime 
combines  at  least  three  strategies.  The  first  is  surge  capability,  the 
ability  to  push  more  out  of  existing  production  lines  with  overtime 
and  multiple  shifts.  This  is  a  "buy  it  out”  strategy.  The  second  is 
flexible  manufacturing.  The  third  is  mobilization  of  mothballed  ca¬ 
pacity,  a  diversification  strategy.  The  choice  of  which  form  of  manu¬ 
facturing,  when,  in  what  quantities,  and  for  what  munitions  would  be 
made  during  a  mobilization  and  during  the  war  itself,  depending  on 
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what  munitions  were  being  consumed,  which  were  effective  in  combat, 
which  were  cost  effective,  and  so  on. 

The  CLOUT  initiatives  for  Air  Force  spares  and  repair  are  another 
example  of  a  combined  strategy:  They  involve  lateral  repair  and 
resupply  within  a  theater  of  operations  (greater  scale),  current 
information  about  inventories  and  planes  awaiting  parts  (buying 
more  information),  and  adaptively  scheduling  repair  and  distribution 
to  maximize  aircraft  availability  (procrastinate  and  react). 

Passive  and  Active  Strategies  Compared 

The  foregoing  discussion  was  not  an  attempt  to  prescribe  precise 
solutions  for  any  problem,  but  a  collection  of  analogies  to  make  a  few 
points.  One  central  feature  of  strategies  for  uncertain  situations  is 
clear:  The  cost  and  effectiveness  of  the  overhead  needed  for  an  active 
system  must  be  balanced  against  the  cost  and  effectiveness  of  buying 
amply  for  a  passive  system. 

On  the  cost  side,  the  main  burden  of  a  passive  strategy  is  buying 
enough  “stuff,"  whatever  that  happens  to  be  in  the  case  at  hand. 
Overhead  is  fairly  simple  and  thus  cheap.  Management  is  clearly  a 
more  important  resource  in  diversification  strategies  than  in  “buy  it 
out”  strategies,  but  only  for  the  initial  decisions  on  whether  and  how 
to  diversify;  but  in  active  strategies,  management  is  the  most  impor¬ 
tant  resource.  It  is  needed  to  recognize  key  events  and  adapt  to  them, 
and  it  costs:  Overhead  is  more  elaborate  and  expensive,  particularly 
for  information  systems. 

On  the  effectiveness  side,  the  ampleness  of  a  passive  strategy  is 
wonderful  when  it  can  be  had,  but  it  must  be  possible  to  define  am¬ 
pleness,  to  attain  it,  and  to  afford  it.  For  example,  when  airbase  at¬ 
tacks  can  destroy  repair  capacity  and  stocks  of  parts,  no  amount  of 
extra  parts  at  the  base  will  suffice  to  cover  this  possibility.  In  this 
case,  ampleness  cannot  even  be  defined.  In  other  cases,  ampleness 
can  be  defined  but  it  cannot  be  attained.  It  would  be  desirable,  for 
example,  if  every  aircraft  mechanic  could  perform  every  repair  task, 
but  a  mechanic  can  learn  only  so  many  specialties.  Even  when  am¬ 
pleness  can  be  defined  and  attained  in  principle,  it  may  be  too  expen¬ 
sive:  Uncertainty  may  be  so  great  that  excessive  quantities  must  be 
bought  to  reach  an  acceptable  level  of  confidence. 

If  ampleness  cannot  be  achieved  for  any  of  these  reasons,  active 
strategies  may  be  more  effective.  An  active  system  must  be  able  to 
recognize  key  events,  compute  an  advantageous  reallocation  of  re¬ 
sources,  and  execute  it.  If  any  of  these  capabilities  is  impaired,  or  if 
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there  are  no  key  events  to  recognise  or  no  advantageous  reallocations, 
the  effectiveness  of  an  adaptive  strategy  is  impaired. 

These  generic  strategies  may  be  useful  as  a  source  of  ideas  for  ana¬ 
lysts  designing  possible  systems.  Even  more  useful  would  be  finding 
out  what  actual  managers  do  to  adapt  in  real  situations.  History  of¬ 
fers  an  abundance  of  real  managers  in  real  wars.2  In  addition,  exer¬ 
cises  are  run  now  in  the  armed  forces;  these  could  be  studied,  or  ex¬ 
ercises  could  be  run  specifically  to  stress  organizations  and  observe 
how  managers  adapt.3  Finally,  analysts  can  go  watch  managers  in 
their  day-to-day  activities.  The  adaptive  repair  and  distribution  poli¬ 
cies  analyzed  in  the  CLOUT  study  would  extend  and  formalize  simi¬ 
lar  things  that  Air  Force  managers  do  now,  in  peacetime. 

IMPLICATIONS  FOR  SYSTEM  EVALUATION:  SENSITIVITY 
ANALYSIS  IS  NOT  ENOUGH 

Like  any  other  system,  a  responsive  system  must  be  evaluated  and 
compared  with  alternatives,  and  an  acknowledgment  of  uncertainty 
and  adaptation  has  implications  for  evaluation.  Here,  we  argue  that 
more  use  of  such  standard  tools  as  probability  and  sensitivity  analysis 
will  be  helpful  in  some  situations,  but  those  situations  occur  infre¬ 
quently.  In  other  situations,  a  different  kind  of  logic  and  analysis  is 
needed  to  connect  the  arithmetic  in  the  analysis  to  the  actual  prob¬ 
lem. 

What  Standard  Tools  Can  Do 

Analysts  may  be  able  to  use  familiar  tools  more  often  to  take  better 
account  of  uncertainty.  It  may  be  possible  to  accurately  represent 
statistical  uncertainty  in  models  by  replacing  some  constants  with 
probability  distributions  and  to  represent  some  management  adapta¬ 
tions  in  models.  For  instance,  in  the  Air  Force  spares  example,  it  is 
possible  to  replace  the  Poisson  failure  model  with  a  negative  binomial 
model  that  has  variability  more  like  that  observed  in  peacetime  and 
to  represent  cannibalization  of  parts  from  disabled  aircraft.  It  may 
also  be  possible  to  use  more  sensitivity  analysis  for  uncertain  things 
about  which  probability  distributions  are  difficult  to  specify  or  justify. 

*For  example,  tee  “War  of  the  Accountant#,'  in  Van  Creveld  (1077). 

^pp  (1988),  for  example,  deacribea  the  Air  Force'*  Coronet  Warrior  exerciae,  in 
which  the  Dyna-METRIC  model  waa  teated.  Among  other  thinga,  logiatica  personnel 
were  observed  and  found  to  be  more  innovative  and  to  perform  better  than  predicted. 
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In  the  spares  example,  different  levels  of  airbase  damage  can  be 
examined  in  a  sensitivity  analysis  (and  were  in  the  CLOUT  analysis). 

These  standard  textbook  devices  will  be  useful  and  adequate  when 
the  model  of  statistical  uncertainty  and  management  adaptations 
have  a  firm  grounding  in  fact  and  when  the  things  to  be  varied  in  the 
sensitivity  analysis  can  be  bounded  with  certainty.  In  such  cases, 
plenty  of  tools  exist  and  the  logic  connecting  the  analysis  to  the  world 
is  straightforward:  If  all  the  important  uncertain  things  can  be 
bounded  with  certainty,  and  if  variation  within  those  bounds  does  not 
affect  the  outcome  of  the  analysis,  then  uncertainty  about  those 
things  will  not  upset  the  results  of  the  analysis. 

What  Standard  Tools  Can't  Do 

We  believe  the  situation  just  described  does  not  hold  very  often, 
and  we  are  only  selectively  sanguine  about  the  utility  of  spending  a 
great  deal  of  extra  effort  on  adding  probability  distributions  to  models 
and  doing  dozens  of  sensitivity  runs.  Indeed,  we  doubt  that  it  is  ever 
possible  to  represent  all  the  salient  uncertainty,  although  it  may  be 
possible  to  represent  enough  of  it.  Even  in  operational  analyses,  in 
which  data  are  available,  models  of  statistical  uncertainty  may  be 
difficult  or  impossible  to  justify.  This  is  the  case  in  spare  parts  work. 
Even  for  peacetime,  part  failures  change  from  period  to  period  in 
ways  that  are  difficult  to  reconcile  with  any  handy  statistical  model. 
Further,  the  things  that  are  candidates  for  sensitivity  analyses 
often  cannot  be  bounded  with  anything  resembling  certainty. 
Extrapolations  of  part  failures  to  wartime  are  simply  unknown,  and 
experience  from  earlier  wars  indicates  that  our  predictions  may  be 
wrong  by  orders  of  magnitude. 

One  can  still  spend  considerable  effort  building  more  statistical 
variation  into  models  and  doing  dozens  of  variations  on  them  in  sen¬ 
sitivity  analyses,  and  this  is  commonly  done.  But  without  the  solid 
grounding  for  the  model  of  statistical  uncertainty  and  certain  bounds 
on  the  things  varied  in  the  sensitivity  analysis,  the  analysis  is  with¬ 
out  an  explicit  logical  connection  to  the  world,  and  the  extra  effort  is 
not  of  clear  value. 

So  what  else  can  be  done  when  these  textbook  tools  do  not  do  the 
job?  What  can  fill  the  space  between  the  model/analysis  and  the 
world  about  which  it  is  to  be  informative?  The  space  can  be  filled  in 
logically  clean  and  unclean  ways.  The  term  logically  unclean”  is  not 
meant  to  convey  condemnation;  it  is  meant  to  convey  that  although  a 
logically  unclean  argument  may  be  a  necessary  evil,  it  is  still  unde- 
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sirable.  Logically  unclean  connections  between  a  model/analysis  and 
the  world  are  necessarily  more  subtle  and  elaborate  than  logically 
clean  connections;  we  do  not  understand  them  well  enough  to  pursue 
them  here  and  will  defer  consideration  to  another  study. 

Two  Logically  Clean  Alternatives4 

Two  logically  clean  connections  are  readily  available  and  often 
useful.  The  first  is  to  implement  a  proposed  system,  perhaps  as  a 
prototype,  and  see  if  it  performs  adequately.  As  noted  above,  part  of 
the  CLOUT  system  is  being  prototyped,  and  it  appears  to  perform 
roughly  as  advertised  for  the  peacetime  conditions  in  which  it  is  being 
tested.  Given  that  it  does  perform  adequately,  it  is  almost  irrelevant 
if  the  models  and  assumptions  inside  the  system  are  accurate:  If  the 
system  works,  it  works,  and  it  is  not  necessarily  cost  effective  to  re¬ 
place  inaccurate  models  with  more  accurate  ones. 

Tests  like  this,  however,  are  available  for  only  some  problems,  gen¬ 
erally  for  operational  analyses  involving  adjustments  to  existing  sys¬ 
tems.  Other  problems  may  be  susceptible  to  the  other  logically  clean 
argument,  an  a  fortiori  argument.  For  our  purpose,  a  fortiori  argu¬ 
ments  can  be  described  as  follows:  If  condition  X  were  true,  then  pol¬ 
icy  A  would  be  preferable  to  the  other  candidates,  but  the  actual  situ¬ 
ation  deviates  from  X  in  specific  ways  that  favor  policy  A  even  more, 
so  a  fortiori  policy  A  is  preferable.  The  CLOUT  analyses  are  ani¬ 
mated  by  such  an  argument.  The  model  used  in  these  analyses  treats 
part  failures  as  more  predictable  than  they  really  are  and  allows 
fewer  management  adaptations  than  are  likely  to  be  available.  In 
this  world,  the  CLOUT  initiatives  pay.  In  an  actual  war,  conditions 
will  probably  be  more  favorable  to  CLOUT,  so  a  fortiori  CLOUT  ap¬ 
pears  to  prevail  over  the  current  arrangement. 

The  argument  has  three  parts — that  X  implies  A  is  preferable,  that 
X  represents  a  boundary  on  the  actual  situation,  and  that  deviations 
from  X  will  favor  A  Each  part  of  the  argument  is  essential  and  may 
force  certain  things  on  analysts  using  it.  It  may  be  possible  to  assert 
that  “X  implies  A  is  preferable”  only  by  means  of  a  model.  The  intro¬ 
duction  of  the  model  may,  because  of  its  unrealism,  undermine  the 
assertion,  but  it  need  not.  For  example,  the  Army’s  JANUS  combat 
simulation  model  has  been  attacked  because  of  many  patently  unreal- 


4Neither  of  these  alternatives  ia  our  invention,  or  even  new.  We  emphasize  them  to 
focus  attention  on  the  logic  of  models  and  analysis,  especially  in  the  face  of  pervasive 
uncertainty.  Our  emphasis  on  the  logic  of  the  argument  is  unusual  if  not  novel. 
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istic  features  of  its  simulation  of  the  behavior  of  individual  weapons, 
sensors,  and  their  operators.  But  JANUS  can  still  be  used  to  drive  an 
a  fortiori  argument.  Specifically,  JANUS  is  a  benign  environment  for 
notional  systems:  They  are  operated  properly  and  work  as  adver¬ 
tised,  they  are  generally  not  subjected  to  countermeasures  designed 
specifically  for  them,  and  they  are  not  subject  to  Murphy’s  Law. 
Thus,  if  proposed  systems  don’t  pay  in  JANUS,  a  fortiori  they  will  not 
pay  in  real  combat6 

It  may  not  be  possible  to  find  a  set  of  bounding  conditions  X  tha„ 
permits  an  a  fortiori  argument.  For  example,  many  have  argued  that 
models  simulating  the  European  theater  of  war  are  so  complex  that  a 
set  of  bounding  conditions  is  too  much  to  ask  for.  Indeed,  some  ar¬ 
gue6  that  the  Defense  Department’s  reliance  on  a  fortiori  arguments 
using  worst-case  taskings  for  Europe  are  so  extreme  that  they  conceal 
useful  and  feasible  gains.  Further,  it  may  be  a  matter  of  contention 
how  actuality  differs  from  X  and  whether  the  deviations  favor  A 
Some  have  attacked  the  CLOUT  analysis  with  the  assertion  that 
wartime  will  not  necessarily  be  more  favorable  to  CLOUT,  because  of 
degradations  in  the  necessary  command,  control,  and  information 
systems.  This  is  a  real  concern  and  the  subject  of  current  study. 
Finally,  if  an  acceptable  model  relating  X  to  outcomes  is  available,  it 
may  be  necessary  to  use  it  to  understand  whether  possible  deviations 
from  X  will  indeed  favor  A.7 

IMPLICATIONS  FOR  SETTING  THE  TASKING 

A  tasking  can  serve  several  purposes.  It  may  represent  an  actual 
prediction  of  conditions,  or  it  may  serve  only  to  constrain  the  analyses 
of  claimants  in  a  budget  exercise.  In  either  role,  uncertainty  can  be 
salient  and  can  affect  the  form  the  tasking  takes.  If  the  tasking  rep¬ 
resents  an  attempt  to  predict,  then  generally  it  should  represent 
uncertainty.  It  may  do  this  with  multiple  sets  of  responsibilities  or 
conditions,  whose  use  will  depend  on  the  nature  of  the  analysis  they 
feed.  If  the  tasking  is  a  constraint  on  the  analyses  of  budget 
claimants,  it  can  still  consist  of  several  sets  of  responsibilities  or  con¬ 
ditions,  with  the  injunction  that  the  budget  claims  should  perform 
adequately  against  each  part  of  the  tasking. 

^This  observation  ia  due  to  Richard  Salter. 

®See,  for  example,  Davis  (1988). 

7In  general  it  will  be  eaaier  to  construct  a  fortiori  arguments  in  operational  analyses 
than  in  other  kinds  of  analysis,  because  fewer  features  of  the  problem  are  subject  to 
change,  and  the  behavior  of  the  operational  system  should  be  more  clearly  understood. 
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We  have  few  specific')  to  offer  now  about  selecting  the  multiple  re¬ 
sponsibilities  and  conditions  for  a  tasking.  We  conjecture  that 
progress  will  come  from  the  idea  that  the  tasking  should  be  set  up  to 
drive  a  fortiori  arguments.  It  may  be  possible  to  use  a  tasking  to  cre¬ 
ate  incentives  for  the  tasked  organization  to  institutionalize  man¬ 
agement  adaptations.  For  example,  the  Air  Force  Logistics  Command 
might  task  its  subordinate  organizations  with  (say)  five  responsibili¬ 
ties/conditions,  with  the  objective  of  meeting  a  minimum  level  of  air¬ 
craft  availability  in  each,  and  with  a  kicker:  If  the  subordinate  orga¬ 
nization  can  implement  an  adaptive  procedure  and  demonstrate  a 
saving  in  current  operations,  the  organization  keeps  a  percentage  of 
the  saving.  We  mention  this  example  not  because  it  is  necessarily 
practicable,  but  only  to  suggest  the  possibilities  of  further  work  on 
setting  (askings 
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